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EXECUTIVE  SUMMARY 


This  demonstration  report  describes  the  data  collection,  processing  and  analysis  of  cued- 
interrogation  data  collected  with  Sky  Research’s  SKY3D  vehicular  platform  at  the  Pole 
Mountain  Target  and  Maneuver  Area  (PMTMA),  near  Laramie,  WY.  SKY3D  integrates  a 
MetalMapper  sensor,  real-time-kinematic  global  positioning  system  and  inertial  measurement 
unit  (in  the  form  of  the  Novatel  SPAN)  and  a  custom  user  interface  on  a  Kubota  all-terrain- 
vehicle.  Cued  interrogation  data  were  collected  by  dwelling  for  30  seconds  over  2,370 
anomalies  previously  identified  via  a  full-coverage  geophysical  survey  of  part  of  the  Bisbee  Area 
of  PMTMA. 

The  data  collection  was  conducted  in  the  field  over  a  14  day  period  in  July  and  August  2011. 
Production  rate  was  as  high  as  39  points  per  survey  hour  with  a  maximum  of  326  anomalies 
visited  in  a  day.  During  the  first  eight  field  days,  an  electrical  issue  with  the  instrument 
orientation  sensor  resulted  in  a  lower  average  production  rate  of  2 1  points  per  hour.  During  the 
last  six  field  days,  and  after  the  orientation  sensor  problem  was  rectified,  the  production  rate 
increased  64%  to  an  average  of  32  points  per  hour.  Weather  and  summer  thunderstorms 
impacted  two  days. 

In  this  demonstration  report  we  evaluate  eight  identified  performance  metrics  for  the  technology 
including  four  pertaining  to  the  data  collection  (reliability/robustness,  survey  rate,  percentage  of 
site  covered,  MetalMapper  sensor  position  accuracy)  and  four  to  the  subsequent  processing  and 
classification  of  the  collected  data  (percentage  of  munitions  correctly  identified,  reduction  in 
false-alarm  rate,  appropriate  specification  of  stop-dig  point  and  minimization  of  “can’t  analyze” 
anomalies).  Six  of  the  eight  performance  objectives  were  easily  achieved,  while  one  objective, 
90%  of  MetalMapper  positions  within  30  cm  of  each  Target  of  Interest  (TOI),  was  missed  by  1 
cm  (90th  percentile  at  31  cm).  The  last  and  only  qualitative  objective  on  reliability/robustness 
was  only  partially  met  due  to  the  electrical  issues  with  the  orientation  sensor  on  the  Novatel 
SPAN. 

At  the  final  stop-dig  point  selected  after  analysis  of  the  MetalMapper  data,  369  anomalies  were 
excavated  and  all  160  TOI  were  identified.  A  total  of  209  of  the  2210  clutter  items  were 
excavated:  this  meant  that  91%  of  the  clutter  could  have  been  left  in  the  ground,  resulting  in 
significant  potential  cost  savings.  During  the  Pole  Mountain  demonstration  MetalMapper  data 
were  collected  at  2,370  anomaly  locations.  Based  on  the  cost  information  collected  as  part  of  this 
demonstration  the  per  anomaly  costs  (excluding  mobilization  and  reporting)  were 

•  $41.24  for  data  collection;  and 

•  $4.08  for  data  processing. 

The  total  cost  per  anomaly  (for  data  collection  and  processing)  was  $45.52.  Using  an  often 
quoted  rule  of  thumb  that  each  excavation  costs  $100,  then  without  deploying  the  MetalMapper 
system  the  excavation  costs  would  have  been  approximately  $237,000.  With  the  MetalMapper 
only  369  anomalies  needed  to  be  excavated  at  a  cost  of  $36,900.  When  combined  with  the 
MetalMapper  mobilization,  data  collection,  processing  and  reporting  costs  of  $168,270  the  cost 
of  clearance  using  the  MetalMapper  was  $205,170:  a  saving  of  $31,830  (or  13%). 
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Comparing  EM61  pick  locations  versus  the  actual  ground-truth  locations  of  TOI  revealed  large 
errors  in  position  and  demonstrated  the  importance  of  fine-tuning  the  cued-interrogation  location 
using  the  (currently  limited  but  functional)  real-time  position  estimation  capabilities  of  the 
MetalMapper.  Nightly  quality  control  of  the  collected  data  was  another  important  determinant  of 
cued-interrogation  data  quality,  with  125  anomalies  recommended  for  recollection. 

The  demonstration  was  conducted  under  project  ESTCP  MR-201160  “Data  collection  with 
vehicular  based  advanced  Electromagnetic  Induction  (EMI)  sensors”  as  part  of  the  wider  ESTCP 
Live  Site  Demonstrations. 
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1. ©INTRODUCTION 

1.1  BACKGROUND 

The  Fiscal  Year  2006  Defense  Appropriation  contains  funding  for  the  “Development  of 
Advanced,  Sophisticated,  Discrimination  Technologies  for  UXO  Cleanup”  in  the  Environmental 
Security  Technology  Certification  Program  (ESTCP).  In  2003,  the  Defense  Science  Board 
observed:  “The  problem  is  that  instruments  that  can  detect  the  buried  unexploded  ordnance 
(UXO)  also  detect  numerous  scrap  metal  objects  and  other  artifacts,  which  leads  to  an  enormous 
amount  of  expensive  digging.  Typically  100  holes  may  be  dug  before  a  real  UXO  is  unearthed! 
The  Task  Force  assessment  is  that  much  of  this  wasteful  digging  can  be  eliminated  by  the  use  of 
more  advanced  technology  instruments  that  exploit  modem  digital  processing  and  advanced 
multi-mode  sensors  to  achieve  an  improved  level  of  discrimination  of  scrap  from  UXO.”  The 
discrimination  potential  of  the  MetalMapper  system  has  been  demonstrated  by  Research  & 
Development  (R&D)  staff  at  the  San  Luis  Obispo  demonstration  site  (Billings  et  ah,  2010)  and  a 
second  demonstration  at  Camp  Butner  (Pasion  et  al.,  2012). 

To  date,  testing  of  these  approaches  has  been  primarily  limited  to  test  sites,  with  only  limited 
application  at  live  sites.  Acceptance  of  discrimination  technologies  requires  demonstration  of 
system  capabilities  at  real  UXO  sites  under  real  world  conditions.  Any  attempt  to  declare 
detected  anomalies  to  be  harmless  and  requiring  no  further  investigation  will  require 
demonstration  to  regulators  of  not  only  individual  technologies,  but  an  entire  decision  making 
process. 

To  build  on  this  initial  R&D  success,  and  ensure  that  the  system  is  deployed  across  a  wide  range 
of  UXO  contaminated  sites,  requires  that  the  system  be  turned  over  to  experienced  production 
personnel.  Sky  Research,  Inc.  (SKY)  has  a  Geophysical  Operations  Department  that  undertakes 
Digital  Geophysical  Mapping  (DGM)  operations  at  multiple  sites  within  the  United  States  (U.S.). 
SKY  has  made  a  commitment  to  bringing  discrimination  technologies  to  production  usage  by 
being  the  first  company  to  purchase  a  MetalMapper  sensor.  This  project  will  provide  an 
assessment  of  the  ease  (or  otherwise)  with  which  the  MetalMapper  technology  can  be 
transitioned  to  production  staff. 

1.2  OBJECTIVES  OF  THE  DEMONSTRATION 

The  demonstration  objective  is  to  collect  high  quality  MetalMapper  sensor  data  in  a  cued 
interrogation  mode  over  2,370  anomalies  previously  detected  during  an  EM61  survey. 

The  data  collection  is  intended  to  meet  the  following  objectives: 

1.  Demonstrate  the  transition  of  the  new,  but  now  commercially  available,  MetalMapper 
sensor  to  personnel  engaged  in  production  data  collection  (and  not  involved  in  research 
and  development). 

2.  Provide  advanced  EMI  sensor  data  from  the  MetalMapper  to  a  wide  range  of  data 
analysts  to  test  different  processing  and  interpretation  methodologies. 
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2.0TECHNOLOGY  DESCRIPTION 


EMI  is  generally  considered  to  be  the  most  promising  technology  for  discriminating  between 
UXO  and  non-UXO  items.  In  the  EMI  method,  a  time  varying  field  illuminates  a  buried, 
conductive  target.  Currents  induced  in  the  target  then  produce  a  secondary  field  that  is  measured 
at  the  surface.  EM  data  inversion  involves  using  the  secondary  field  generated  by  the  target  for 
recovery  of  the  position,  orientation,  and  parameters  related  to  the  target’s  material  properties 
and  shape.  In  the  UXO  community,  the  inverse  problem  is  usually  simplified  by  assuming  that 
the  secondary  field  can  be  accurately  approximated  as  a  dipole. 

Time-domain  Electromagnetic  (TEM)  sensors  illuminate  a  buried  target  by  rapidly  turning  off  a 
transmitting  loop  which  causes  a  step-change  in  the  magnetic  field  below  the  ground,  inducing 
eddy  currents  in  any  metallic  objects.  The  eddy  currents  induced  in  the  target  decay  with  time, 
generating  a  decaying  secondary  field  that  is  measured  at  the  surface.  The  time-varying 
secondary  magnetic  field  B(t)  at  a  location  r  from  the  dipole  m(t)  is: 

B(t)  =  4^m(t) ' (3ff  “  f) 

Where  r  =  r/\r\  is  the  unit- vector  pointing  from  the  dipole  to  the  observation  point,  I  is  the  3  x 
3  identity  matrix,  /r0  =  4  n  x  10-7  H/m  is  the  permittivity  of  free  space  and  r  =  |r|  is  the  distance 
between  the  center  of  the  object  and  the  observation  point. 

The  dipole  induced  by  the  interaction  of  the  primary  field  B0  and  the  buried  target  is  given  by: 

1  _ 

m(t)  =  — M(t )  ■  B0 
Mo 


where  M(t)  is  the  target’s  polarization  tensor.  The  polarization  tensor  governs  the  decay 
characteristics  of  the  buried  target  and  is  a  function  of  the  shape,  size,  and  material  properties  of 
the  target.  The  polarization  tensor  is  written  as: 


M(t)  = 


Li  (0 
0 
0 


0 


L2(t) 

0 


0 

0 


i3(0. 


where  we  use  the  convention  that  >  L2(U)  ^  L3(tt)  so  that  polarization  tensor 

parameters  are  organized  from  largest  to  smallest.  The  polarization  tensor  components  are 
parameterized  such  that  the  target  response  can  be  written  as  a  function  of  a  model  vector 
containing  components  that  are  a  function  of  target  characteristics. 


ESTCP  sponsored  discrimination  pilot  projects  at  the  Former  Camp  Sibert  and  San  Luis  Obispo 
have  demonstrated  the  ability  of  advanced  EMI  sensors  to  constrain  the  polarization  tensor 
parameters  of  buried  metallic  objects  (e.g.  Billings  et  al.,  2008,  2010).  Attributes  such  as  the 
size,  decay  rate  and  symmetry  (or  lack  thereof)  of  the  polarization  tensor  parameters  provide  an 
indication  of  the  identity  of  the  underlying  object.  Intelligent  classification  techniques  have 
demonstrated  an  impressive  ability  to  produce  a  prioritized  dig-list  with  most,  or  all,  of  the 
Targets  of  Interest  (TOI)  ranked  as  high-priority  excavations. 
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2.1  METALMAPPER  SENSOR 

In  2010,  SKY  purchased  the  first  commercially  available  MetalMapper  sensor.  The 
MetalMapper  is  a  3 -axis  transmit,  3 -axis  receive  EMI  array  developed  by  G&G  Sciences  and 
commercially  produced  by  Geometries.  This  array  has  been  incorporated  into  SKY’s  UXO 
discrimination  system  known  as  the  SKY3D  (Figures  1  and  2),  which  comprises  an  all-terrain 
survey  vehicle,  the  sensor  frame  and  associated  suspension,  the  MetalMapper  advanced  sensor, 
global  positioning  system  (GPS)  and  inertial  measurement  unit  (IMU)  sensors  for  precise 
positioning,  data  acquisition  hardware  (National  Instruments  PXI-1031)  and  software,  and 
discrimination  processing  software  modules.  The  system  completed  its  first  field  trial  during  the 
summer  of  2010  at  the  ESTCP  Live  Site  Demonstration  at  Camp  Butner  in  North  Carolina. 

The  MetalMapper  sensor  head  consists  of  three  orthogonal  1 -meter  square  transmitter  loops  that 
excite  nearby  targets  with  a  large-moment  magnetic  field.  The  sensor  base  contains  an  array  of 
seven  small  3-axis  receiver  cubes  that  measure  the  complete  vector  magnetic  field  at  an 
optimized  distribution  of  measurement  locations.  Figure  3  shows  the  layout  of  the  receiver 
cubes. 


Figure  1.  The  SKY3D  MetalMapper.  The  SKY3D  MetalMapper  extended  in  survey  mode  (LEFT) 

and  elevated  in  transport  mode  (RIGHT). 


Figure  2.  A  Close-Up  of  a  MetalMapper  Receiver  Cube 
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Figure  3.  Distribution  of  Receiver  Locations  in  the  MetalMapper  Sensor  Base. 


Each  receiver  measures  the  complete  vector  magnetic  field  at  its  location  to  fully  characterize  the 
target  response  to  the  impinging  transmitter  field. 


The  MetalMapper  can  be  deployed  in  either  “dynamic”  mode  or  in  a  “cued-interrogation”  mode. 
In  the  former,  data  are  collected  continuously  while  the  vehicle  traverses  over  an  area.  This 
mode  is  primarily  intended  for  detection  and  utilizes  only  the  z-axis  transmitter.  Each  detection 
(or  a  subset  of  detections  if  prescreening  is  applied)  is  then  revisited  in  turn  with  the  sensor 
operating  in  the  second,  cued-interrogation,  mode.  The  array  is  approximately  centered  over  the 
unknown  object  and  data  are  collected  with  the  system  static.  In  this  mode,  all  three  orthogonal 
transmitters  are  used,  with  data  collected  after  excitation  of  the  ground  by  each  transmitter.  Note 
that  the  MetalMapper  could  also  be  cued  by  an  alternative  detection  sensor  such  as  an  EM61. 


The  combination  of  orthogonal  transmit  and  receive  data  yields  an  extremely  rich  set  of  feature 
vectors  that  can  be  used  to  characterize  an  anomaly.  Specifically,  the  3-axis  illumination  ensures 
a  more  accurate  recovery  of  target  polarizability  parameters  than  that  afforded  by  vertical  axis 
illumination  alone.  Discrimination  is  achieved  when  an  anomaly  can  be  confidently  classified  as 
either  a  TOI  or  a  non-TOI.  Usually  this  decision  is  based  on  statistical  classification  of  anomaly 
features  as  either  target-like  or  non-target-like.  The  determination  of  TOI  characteristics  versus 
non-TOI  characteristics  is  achieved  through  analysis  of  training  data,  which  comprise  the 
responses  of  a  sample  set  of  targets  and  clutter  that  are  representative  of  those  expected  to  be  at 
the  survey  site.  Features  for  analysis  may  be  extracted  from  inversion  of  physics-based  forward 
models  or  through  direct  interpretation  of  the  data.  Examples  of  relevant  features  include  the 
shape  of  the  item  as  determined  by  comparison  of  the  primary,  secondary,  and  tertiary 
polarizations;  time  decay  information  based  on  the  decay  of  the  primary  polarization;  and  the 
size  of  the  object  as  determined  by  scaling  parameters  recovered  from  inversion  of  the  forward 
model. 


2.2  TECHNOLOGY  MATURITY 

The  MetalMapper  is  a  mature  technology  that  has  been  tested  at  the  standardized  UXO  test-sites 
as  well  as  at  the  San  Luis  Obispo  and  Camp  Butner  demonstration  sites.  The  SKY  MetalMapper 
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was  deployed  at  the  Camp  Butner  site  and  was  used  to  collect  about  half  of  the  cued- 
interrogation  anomalies  that  were  surveyed  at  that  site.  Analysis  of  the  data  from  Camp  Butner 
by  Dr.  Donald  Snyder  revealed  that  the  SKY  incarnation  of  the  MetalMapper  had  superior 
Signal-to-noise  Ratio  (SNR)  characteristics  compared  to  the  original  Geometries  deployed 
system:  SNR  was  improved  by  around  10  decibels  (dB).  There  were  a  number  of  contributing 
factors  for  this  improvement: 

1 .  The  SKY  system  was  7  centimeters  (cm)  closer  to  the  ground  than  the  Geometries  system 

2.  The  newer  preamplifiers  and  electronics  in  the  SKY  system  have  lower  noise  than  the 
original  system. 

3.  The  SKY  vehicle  was  turned  off  during  data  acquisition  while  the  Geometries  vehicle 
was  left  running. 

4.  The  Geometries  deployed  system  suffered  from  periodic  malfunction  of  two  of  the 
receivers. 

A  number  of  enhancements  were  made  to  the  MetalMapper  sensor  after  the  Camp  Butner 
deployment  including: 

•  Modified  power  harness  to  provide  alternating  current  (AC)  and  direct  current  (DC) 
options  and  polarity  reversal  protection.  The  Data  Acquisition  System  (DAQ)  and 
receiver  cubes  are  powered  through  an  inverter  running  off  two  marine  deep  cycle 
batteries  coupled  in  parallel.  The  transmitters  are  supplied  by  up  to  four  internal  Ultralife 
batteries  coupled  in  parallel.  This  power  configuration  has  enough  capacity  for  almost 
two  days  of  cued  interrogation. 

•  Mounted  all  electronics  (DAQ,  inverter,  batteries,  cables,  etc.)  in  Kubota  bed  using 
waterproofed  plywood  palette  and  mounting  brackets. 

•  Installed  vibration  damping  material  in  the  sensor  head  mount  to  reduce  vibration 
induced  noise. 

•  Replaced  winch  cable  with  higher  strength  Amsteel  Blue  cable. 

•  Reinforced  winch  cable  tower  with  welded  gussets  to  reduce  strain  on  assembly  when 
sensor  head  is  suspended. 

•  Routed  all  cables  through  plastic  conduit  mounted  on  sensor  frame  arms. 

•  Improved  cab  mounts  for  monitor,  keyboard,  and  mouse. 

•  Reinforced  sensor  head  tray  assembly  and  coated  with  spar  urethane. 

2.2.1  Positioning  System:  Novatel  SPAN 

A  Novatel  SPAN  system  combines  a  GPS  receiver  with  an  Inertial  Measurement  Unit  (IMU). 
An  accurate  3D  position,  velocity  and  attitude  information  is  continuously  output  from  the 
SPAN,  even  when  GPS  information  becomes  unavailable.  A  Novatel  ProPak  v3  receiver  was 
mounted  in  the  bed  of  the  tow  vehicle  with  the  Novatel  IMU-G2-000  IMU  and  GPS  antenna 
mounted  above  the  center  of  the  sensor  coils.  Position  data  was  output  in  a  NMEA  format  at  a 
rate  of  20Hz.  In  cued  interrogation  mode,  one  position  value  was  recorded  per  cued  point. 
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Figure  4  Sky  Research  utilizes  the  Novatel  Span  System. 

This  device  provides  position,  pitch,  heading  and  roll  information. 

2.2.2  Data  Acquisition  System 

The  data  acquisition  system  (DAQ)  is  built  around  a  commercially  available  product  from 
National  Instruments.  The  DAQ  and  EM  transmitter  are  packaged  in  an  aluminum  case  that 
weighs  approximately  43  pounds  when  all  four  transmitter  batteries  are  installed. 

The  DAQ  is  a  full-featured  PC  running  Windows  7.  It  contains  disk  storage,  serial  and  USB 
input/output  ports,  and  more.  It  is  interfaced  to  analog-to-digital  converters  and  to  digital 
input/output  devices  through  its  internal  PCI  bus.  It  is  packaged  in  an  industry  standard  PXI 
configuration  that  is  intended  for  industrial  applications.  Figure  5  is  a  functional  block  diagram 
of  the  DAQ  instrument  package. 

2.3  PREVIOUS  TESTING  OF  THE  TECHNOLOGIES 

The  MetalMapper  sensor  system,  using  a  different  Novatel  SPAN,  was  used  by  Sky  Research  at 
San  Luis  Obispo  and  Camp  Butner  demonstration  sites. 

2.4  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGIES 

The  main  limitations  of  the  MetalMapper  sensor  system  in  a  cued-interrogation  mode  relate  to 
the  equipment  and  labor  costs.  Cued-interrogation  introduces  additional  costs  in  that  each 
selected  anomaly  has  to  be  visited  with  a  second  geophysical  survey  system.  The  MetalMapper 
system  also  has  a  limited  range  of  terrain/vegetation  that  can  be  accessed  by  the  system. 
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The  additional  labor  and  equipment  costs  may  be  offset  by  reducing  the  amount  of  digs 
preformed  by  the  UXO  intrusive  team. 


Figure  5 


A)  -The  MetalMapper  (DAQ)  and  B) 


Functional  Block  Diagram 
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3.0PERFORMANCE  OBJECTIVES 

Performance  objectives  for  the  demonstration  are  given  in  Table  1  to  provide  a  basis  for 
evaluating  the  performance  and  costs  of  the  demonstrated  technology.  These  objectives  are  for 
the  technology  being  demonstrated  only;  overall  project  objectives  will  be  given  in  the 
demonstration  report  to  be  prepared  by  ESTCP. 

Performance  metrics  of  attributes  derived  from  the  collected  data  (e.g.  discrimination  results) 
will  determine  whether  this  demonstration  ultimately  meets  its  desired  objective  of  successful 
discrimination  performance  for  the  MetalMapper  cued-interrogation  system. 


Table  1  Performance  Objectives/Metrics  and  Confirmation  Methods 


Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Result  achieved 

Reliability 

and 

robustness 

Operator 

feedback 

•  Field  notes 

Operator  acceptance 

Partially  MET 

Some  issues  with 
Novatel  SPAN 

Survey  rate 

Number  of 
targets 
visited  per 
day 

•  Field  notes 

More  than  200  per 
day 

MET 

204  per  day 

215  per  day  (excl 
short  days) 

Percent 

completed 

Percentage 
of  anomalies 
visited 

•  Field  notes 

Cued  data  obtained 
for  all  EM61  target 
picks 

MET 

100%  of  anomalies 
surveyed 

Location 

accuracy 

Distance 
between 
center  of 
sensor  and 
anomaly 

•  GPS  location 
in  collected 
groundtruth 

•  MM  position 
and  IMU 

MM  center  location 
within  30  cm  of  TOI 
at  least  90%  of  the 
time 

MET 

(to  within  1  cm) 

90%  within  3 1  cm 

Maximize 

correct 
classification 
of  munitions. 

Number  of 
targets-of- 
interest 
retained. 

•  Prioritized 
anomaly 
lists, 

•  Scoring 
reports  from 
IDA, 

Approach  correctly 
classifies  all 
targets-of-interest. 

MET 

Pd  =  100% 

Maximize 

correct 

classification 

of 

non¬ 

munitions. 

Number  of 
false  alarms 
eliminated. 

•  Prioritized 
anomaly 
lists, 

•  Scoring 
reports  from 
IDA, 

Reduction  of  false 
alarms  by  >75% 
while  retaining  all 
targets  of  interest. 

MET 

91%  reduction  in  FA 
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Performance 

Objective 

Metric 

Data  Required 

Success  Criteria 

Result  achieved 

Specification 

of 

no-dig 

threshold. 

Pclass  and  Nfa 

at 

demonstrator 

operating 

point. 

•  Demonstrator 
-specified 
threshold. 

•  Scoring 
reports  from 
IDA. 

Threshold  specified 
by  the  demonstrator  to 
achieve  criteria  above. 

MET 

Pd  =  100% 

91%  reduction  in  FA 

Minimize 
number  of 
anomalies 
that  cannot 
be  analyzed. 

Number  of 
anomalies 
that  must  be 
classified  as 
“Unable  to 
Analyze.” 

•  Demonstrator 
target 

parameters. 

Reliable  target 
parameters  can  be 
estimated  for  >  90% 
of  anomalies  on  each 
sensor’s  detection  list. 

MET 

0%  of  anomalies  were 
can’t  analyze 
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4.0SITE  DESCRIPTION 

4.1  SITE  SELECTION 

The  Pole  Mountain  Target  and  Maneuver  Area  is  a  62,448.15  acre  site  located  near  Laramie, 
Wyoming.  The  demonstration  was  conducted  in  the  Bisbee  Hill  Maneuver  Area.  This  site  was 
chosen  as  the  next  in  a  series  of  sites  for  demonstration  of  the  classification  process.  The  first 
site  in  the  series,  former  Camp  Sibert  in  Alabama,  had  only  one  target-of-interest  and  item  “size” 
was  an  effective  discriminate.  A  hillside  range  at  the  former  Camp  San  Luis  Obispo  in 
California  was  selected  for  the  second  of  these  demonstrations  because  of  the  wider  mix  of 
munitions,  including  60-mm,  81 -mm,  and  4.2-in  mortars  and  2.36-in  rockets.  Three  additional 
munitions  types  were  discovered  during  the  course  of  the  demonstration.  The  third  site  chosen 
was  the  former  Camp  Butner  in  North  Carolina.  This  site  is  known  to  be  contaminated  with 
items  as  small  as  37-mm  projectiles,  adding  yet  another  layer  of  complexity  into  the  process. 
The  fourth  site,  the  former  Mare  Island  Naval  Shipyard  (MINS)  in  Vallejo,  CA,  was  selected 
because  of  an  opportunity  in  the  Navy’s  remediation  schedule  at  MINS  to  conduct  the  study  in 
the  midst  of  their  ongoing  munitions  response  project  and  prior  to  the  upcoming  removal  action 
in  2012.  The  fifth  site,  Camp  Beale  in  Yuba,  CA,  was  selected  for  demonstration  because  it  is 
partially  wooded  and  is  thought  to  contain  a  wide  mixture  of  munitions. 

This  site  was  selected  because  of  its  wide  mixture  of  munitions  and  variable  terrain.  The 
smallest  known  munition  on  the  site  is  the  37-mm  projectile,  the  largest  known  items  are  3 -inch 
projectiles  and  mortars,  with  a  range  of  munition  sizes  in  between. 

Figure  6,  extracted  from  the  ESTCP  Study  Plan  (Figure  4-2  in  (ESTCP,  2011),  outlines  the 
demonstration  site  boundary,  comprising  50  acres.  A  detailed  description  of  the  site,  its  history, 
and  other  useful  details  is  contained  in  the  ESTCP  Study  Plan  (ESTCP,  2011). 

4.2  SITE  HISTORY 

The  PMTMA  was  established  in  1879  as  the  Fort  D.A.  Russell  Wood  and  Water  Reserve.  The 
land  status  alternated  between  national  forest  and  military  reservation  from  1897  to  1925.  The 
Pole  Mountain  area  has  also  been  known  as  the  Crow  Creek  Forest  Reserve,  Fort  D.A.  Russell 
Target  and  Maneuver  Range,  Fort  Francis  E.  Warren  Target  and  Maneuver  Range,  Pole 
Mountain  Reservation,  Pole  Mountain  Training  Annex,  and  Warren  Training  Annex.  It  was 
extensively  used  before  1959  as  a  target  and  maneuver  area  by  the  Army,  the  Reserve  Officers’ 
Training  Corps,  the  Citizens’  Military  Training  Corps,  various  National  Guard  units,  and  the 
Department  of  the  Air  Force. 

4.3  MUNITIONS  CONTAMINATION 

A  large  variety  of  munitions  have  been  reported  as  used  at  PMTMA.  Physical  evidence  for  the 
following  items  was  discovered  during  the  RI: 

•  Projectiles  containing  high  explosive  (HE)  filler  (37-mm  to  155-mm,  and  2.95-inch). 

•  Shrapnel  projectiles  (75-mm  and  3-inch). 
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37-mm  projectiles  (inert  and  unfuzed). 

3 -inch  Stokes  mortars  (practice,  fuzed). 

60-mm  mortars  containing  HE  filler. 

Small  arms  ammunition  (.30-caliber  and  .50-caliber). 


469,800  470,000  470,200  470,400  470,600 


UTM  Easting  (M) 

Figure  6  Final  Demonstration  Site  Boundary  (50  acres) 
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4.4  SITE  GEODETIC  CONTROL  INFORMATION 

The  survey  monuments  in  Table  3  were  used  for  setting  up  the  GPS  base-station  and  verifying 
GPS  rover  position.  Sky  Research  established  the  control  points  CP3  and  CP4. 


Table  2  List  of  Primary  Survey  Monuments 


(Used  for  the  surveying  in  UTM-Zone  16N,  NAD-83) 


Survey 

Monuments 

Easting  (m) 

Northing  (m) 

Elevation 

(m) 

ESTCP 1 

4566072.102 

471029.953 

2454.650 

3.5"  ALUM 
DISC  IN 
CONC. 

Published  value 

ESTCP2 

4566115.924 

471090.253 

2448.180 

3.5"  ALUM 
DISC  IN 
CONC. 
"ESTCP2 
2010" 

Published  value 

CP3 

4566553.883 

468904.088 

2524.249 

1/8"  RBR  1" 
PROUD  OF 
GROUND 
SURFACE 

Derived  from  1  hour 
GPS  observation  post 
processed  through 
NGS  OPUS 

CP4 

4566537.402 

468949.922 

2523.396 

TENT 

STAKE 
W/CTR 
PUNCH 
ADJACENT 
TO  TEST 

PIT 

Derived  from  1  hour 
GPS  observation  post 
processed  through 
NGS  OPUS 
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5.0TEST  DESIGN 

5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

The  objective  of  this  program  is  to  demonstrate  a  methodology  for  the  use  of  classification  in  the 
munitions  response  process.  The  three  key  components  of  this  methodology  are  collection  of 
high-quality  geophysical  data  and  principled  selection  of  anomalous  regions  in  those  data, 
analysis  of  the  selected  anomalies  using  physics-based  models  to  extract  target  parameters  such 
as  size,  shape,  and  materials  properties,  and  the  use  of  those  parameters  to  construct  a  prioritized 
dig  list.  This  demonstration  report  predominantly  addresses  the  data-collection  aspect  of  the  Pole 
Mountain  demonstration,  specifically  the  collection  of  cued-interrogation  data. 

5.2  PRE-DEMONSTRATION  ACTIVITIES 

The  main  ESTCP  demonstration  report  for  Pole  Mountain  (ESTCP,  2011)  lists  all  the  pre¬ 
demonstration  activities  conducted  at  the  site  including: 

•  Collection  of  historical  records  about  the  site  through  coordination  with  the  Omaha 
District,  US  Army  Corps  of  Engineers 

•  EM61  transects  to  define  initial  demonstration  area 

•  Establishment  of  two  first  order  navigation  points  to  be  used  for  all  emplacement,  data 
collection,  and  validation  activities. 

•  Surface  clearance  of  the  site 

•  Development  of  a  seed  plan 

•  Establishment  of  an  instrument  verification  strip  near  the  demonstration  area  (Table  3). 

Prior  to  mobilization  to  Pole  Mountain,  we  spent  approximately  one  week  collecting  cued- 
interrogation  data  over  a  test  plot.  This  “shake-down  test”  allowed  the  field  crew  to  become 
more  familiar  with  the  operation  of  the  different  instruments  and  in  the  correct  operating 
procedure  for  the  cued-interrogation  surveys. 

5.3  DATA  COLLECTION  PROCEDURES 
5.3.1  Demonstration  Set-Up  and  Start-Up 

Sky  Research  deployed  to  Pole  Mountain  on  July  20,  2011,  and  unpacked,  assembled,  and 
function  checked  equipment.  Data  collection  (two  cued  points)  started  that  day.  The  following 
general  procedures  were  followed  for  each  survey  day: 

•  Morning  brief  and  tailgate  safety  talk. 

•  GPS  base  station  setup,  GPS  rover  position  check. 

•  MetalMapper  setup  and  start-of-day  equipment  tests. 

•  Field  check  of  start-of-day  tests. 

•  Mobilization  of  MetalMapper  to  data  collection  area. 
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•  For  each  cued  point,  the  anomaly  was  surveyed  using  the  procedure  outlined  in  the  next 
section. 

•  Throughout  the  survey  day,  a  static  background  measurement  was  conducted  at  a  nearby, 
metal-free  location. 

•  End-of-day  equipment  tests. 

We  used  a  two  person  field  crew.  One  operated  the  MetalMapper  while  the  other 
processed/uploaded  data  and  monitored  weather  conditions. 


Table  3.  Contents  of  the  Instrument  Verification  Strip. 


Item 

ID 

Description 

Easting  (m) 

Northing  (m) 

Depth 

(m) 

Inclination 

Orientation 

T-001 

Shotput 

4566543.87 

468927.18 

0.30 

N/A 

N/A 

T-002 

Small  ISO 

4566543.77 

468922.15 

0.15 

Horizontal 

Across  Track 

T-003 

Small  ISO 

4566543.67 

468917.36 

0.15 

Horizontal 

Along  Track 

T-004 

37  mm 

4566543.53 

468912.37 

0.15 

Horizontal 

Across  Track 

T-005 

75  mm 

4566543.504 

468907.465 

0.15 

Horizontal 

Across  Track 

5.3.2  Cued  interrogation  procedure 

MetalMapper  data  were  collected  over  2,370  anomaly  locations  provided  by  the  ESTCP  Program 
Office  using  the  acquisition  parameters  listed  in  Table  4.  A  mapping  interface  facilitated 
positioning  of  the  sensor  by  displaying  the  sensor  location  relative  to  the  cued  anomaly 
coordinates.  Once  the  sensor  is  within  a  few  feet,  the  operator  uses  the  real-time  receiver  cube 
response  display  to  center  the  sensor  over  the  object. 

Average  point  to  point  transit  and  sensor  centering  time  varied  from  90  seconds  to  up  to  3 
minutes.  Once  at  the  point,  data  collection  took  less  than  30  seconds. 

The  standardization  and  calibration  tests  described  in  the  next  section  were  conducted  during 
each  day  of  surveying. 

Table  4  Acquisition  Parameters:  nRpts  =  Number  of  repeats,  Win  =  Window,  RxMode  = 

Receiver  Mode,  TxMode  =  Transmitter  Mode 


Mode 

Hold-Off 

Time 

(us) 

Block 

Period 

(s) 

nRpts 

Win 

Width 

(%) 

nStks 

RxMode 

TxMode 

Static 

50 

0.9 

27 

10 

10 

DecayDecimated 

ZYX 

Triggering  the  MetalMapper  transmit  coil  caused  intermittent  resetting  of  the  IMU  heading 
values.  Commencing  on  the  third  day  of  data  collection,  the  heading  from  a  handheld  compass 
was  manually  recorded  as  a  backup  to  the  IMU  heading  measurement. 
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5.3.3  Calibration  activities 

The  following  calibration  activities  were  undertaken  during  the  course  of  the  survey: 

•  Twice-daily  measurements  of  the  IYS. 

•  Background  measurements  in  a  metal  free  area  conducted  at  least  twice  per-day. 

•  Measurements  of  a  number  of  test-objects  in  a  shallow-test  pit  were  completed  at  the  start 
of  the  survey  period.  Each  projectile  was  measured  in  multiple  orientations  and  at  two 
depths  (10  and  20  cm  to  top  of  item).  Items  measured  included: 

o  37  mm  projectile 

o  57  mm  projectile 

o  75  mm  projectile  (also  at  30  cm  depth) 
o  Stokes  mortar  (also  at  30  and  45  cm  depths) 
o  Calibration  ball  (single  measurements  at  10  and  20  cm) 

5.3.4  Quality  checks 

Initial  data  validation  for  each  anomaly  was  performed  on-site  by  a  member  of  the  survey  team. 
This  processing  was  done  to  verify  the  integrity  of  the  data  collected  and  enabled  any  data 
problems  to  be  immediately  found  and  rectified.  If  questionable  data  were  observed,  they  were 
uploaded  to  the  project  FTP  site  for  immediate  analysis  by  the  project  data  processor. 

5.3.5  Period  of  Operation 

The  field  portion  of  this  demonstration  commenced  on  July  20,  2011,  and  was  completed  on 
August  5,  2011.  A  summary  of  activities  on  each  day  are  provided  in  the  table  below,  with  a 
more  detailed  description  provided  in  the  field  notes  accompanying  the  data  deliverable.  The 
points  per  day  in  Table  8  refer  to  the  number  of  points  completed,  and  do  not  include  multiple 
data  shots  due  to  poor  positioning,  multi-peak  anomalies,  etc. 

5.3.6  Demobilization 

At  the  end  of  field  operations,  all  equipment,  materials,  and  supplies  were  removed  from  the  site 
and  returned  to  Sky  Research’s  office  in  Denver,  Colorado. 

5.3.7  Health  and  Safety  Plan 

A  host  organization  exists  for  this  demonstration  site.  All  field  work  was  conducted  under  the 
authority  of  the  existing  work  plan.  No  separate  Health  and  Safety  Plan  was  required. 
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Table  5  Summary  of  On-Site  Activities. 


Day 

Summary  of  activities 

Approximate 

Points/Hr 

July  20,  2011 

Mobilize  equipment  from  Denver,  CO  to  the  Pole 
Mountain  project  site.  Unload,  assemble  and 
function  check  equipment.  Check  GPS  base  station 
and  survey  control.  Collect  test  pit  data  and  begin 
cued  points. 

7 

July  21,  2011 

Troubleshooting  of  Inertial  Navigation  System  (INS) 
alignment  problem.  Collected  limited  cued  data. 

15 

July  22,2011 

Collected  cued  data  over  21 1  points,  questionable 
heading  values. 

24 

July  24, 2011 

Collected  cued  data  over  186  points.  Began 
recording  backup  heading  values  as  measured  on  a 
handheld  compass. 

22 

July  25,  201 1 

Collected  cued  data  over  201  points. 

24 

July  26,  2011 

Collected  cued  data  over  94  points,  stopped  early  due 
to  lightning. 

13 

July  27,2011 

Collected  cued  data  over  154  points.  Repaired 
fairlead  on  vehicle  platform. 

18 

July  28,  2011 

Collected  cued  data  over  209  points,  INS  alignment 
stable  for  part  of  the  day. 

28 

July  31,  2011 

Collected  cued  data  over  162  points,  stopped  early 
due  to  lightning.  INS  alignment  stable  for  most  of 
the  day. 

27 

August  1,  201 1 

Collected  cued  data  over  251  points.  INS  alignment 
stable  for  entire  day. 

28 

August  2,  201 1 

Collected  cued  data  over  247  points.  INS  alignment 
stable  for  entire  day. 

30 

August  3,  2011 

Collected  cued  data  over  214  points,  stopped  early 
due  to  lightning.  INS  alignment  stable  for  entire  day. 

33 

August  4,  201 1 

Collected  cued  data  over  316  points.  INS  alignment 
stable  for  entire  day. 

35 

August  5,  201 1 

Collected  cued  data  over  68  points  and  80  recollects. 
Additional  test  pit  data  collected.  INS  alignment 
stable  for  entire  day. 

39 

August  7,  2011 

Packed  up  equipment,  demobilized  from  project  site 
back  to  Denver,  Colorado. 
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5.4  MANAGEMENT  AND  STAFFING 

The  responsibilities  for  this  demonstration  are  outlined  in  Figure  7.  Dr.  Stephen  Billings  was  the 
Principal  Investigator  (PI)  for  this  project. 


Figure  7  Management  and  Staffing  Diagram 
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6. ©CALIBRATIONS,  DATA  PROCESSING  AND  ARCHIVING 

6.1  PREPROCESSING 

The  following  pre-processing  steps  were  conducted. 

•  Initial  review  of  collected  data:  Validate  that  metadata  (point  ID,  heading)  is  correct. 

•  Calculation  of  orientation:  Calculate  the  orientation  of  the  MetalMapper  sensor  using 
IMU  or  manual  compass  heading. 

•  Background  removal:  Throughout  the  survey  day,  background  data  are  recorded  with  the 
MetalMapper  in  a  metal-free  part  of  the  survey  area.  For  each  recorded  cued  point,  the 
background  recorded  from  the  closest  measurement  (in  time)  was  subtracted  from  the 
MetalMapper  receiver  data. 

•  Data  conversion:  Data  are  converted  into  a  .csv  file  format  for  analysis  in  Geosoft  or 
UXOLab. 

6.2  INITIAL  QUALITY  CONTROL 

Each  night  an  initial  QC  of  the  data  was  conducted  by  a  QC  analyst  to  determine  if  there  were 
any  problems  with  the  data  that  would  require  any  anomalies  to  be  recollected.  The  QC  review 
included  comparing  the  location  of  the  center  of  the  MetalMapper  against  the  location  of  the 
EM61  anomaly  pick.  Errors  of  larger  than  about  a  meter  typically  occurred  when  the 
MetalMapper  measurement  was  associated  in  the  field  notes  with  the  wrong  EM61  anomaly. 
Usually,  this  mis-association  occurred  when  the  MetalMapper  acquisition  was  either  one  point  in 
front  or  behind  the  EM61  anomaly  list,  and  affected  several  anomalies  in  a  row.  The  problem 
was  easily  remedied  by  shifting  the  MetalMapper  data  up  or  down  one  target  in  the  EM61  list. 

Differences  in  location  of  less  than  1  m  were  typically  caused  when  the  MetalMapper  was  not 
centered  correctly  over  the  buried  item  (e.g.  Figure  8).  If  this  occurred,  the  QC  analyst  would 
flag  the  anomaly  for  recollection  and  would  communicate  that  information  to  the  field-team.  A 
total  of  125  anomalies  required  recollection. 

6.3  DAILY  CALIBRATION  RESULTS 

The  QC  analyst  also  analyzed  the  twice-daily  IVS  data  and  compared  the  predicted  and 
measured  locations  of  each  IVS  item  (Figure  9)  as  well  as  the  principle  axis  polarizabilities 
(Figure  10).  During  the  first  two  days,  compass  measurements  were  not  taken  and  the  IMU 
information  was  corrupt  for  a  number  of  IVS  measurements.  This  resulted  in  errors  of  30  cm  (or 
occasionally  larger)  in  the  IVS  positions.  Once  compass  measurements  were  taken  and 
incorporated  in  the  data  processing  the  predicted  IVS  positions  varied  by  less  than  10  cm  day  to 
day  and  were  typically  biased  about  5  to  12  cm  to  the  South-West  (Table  6). 
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(a)  Original  position  anomaly  912 


X=  -  0  .  3  8 
Y=  -  0  .  4  5 

Target  location 


-1  0  1  chi 


(b)  Repeat  position  anomaly  912 


X= -  0  .  1  7 
Y  =  0.06 

Target  location 


-1  0  1  chi 


(c)  Original  acquisition  anomaly  912 


-28.32:6.53  -6.37:14.60  -6.86:27.11  -2.56:5.39  -8.02:5.96  -8.86:9.50  -80.92:21.99  -15.64:20.80  -21.23:72.87 


Figure  8.  Example  application  of  daily  QC  that  identified  a  poorly  centered  MetalMapper 
acquisition.  The  top  row  shows  the  MetalMapper  position  of  the  original  and  recollected  data 
against  the  EM61  anomaly  map.  The  bottom  two  rows  show  the  observed,  predicted  and  residual 
MetalMapper  data  for  the  original  (c)  and  recollected  (d)  locations. 
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Errors  in  predicted  location:  27  measurements 


Figure  9.  AM/PM  Interpreted  Location  of  the  IVS  Items.  Items  with  a  circle  were  collected  without 
valid  IMU  data  and  those  highlighted  yellow  had  no  compass  measurement. 

Table  6.  Summary  of  location  errors  on  the  IVS  items  (excluding  days  without  compass 


measurements). 


Item 

ID 

Description 

RMS  error 
Easting 
(cm) 

RMS  error 
Easting 
(cm) 

Bias 

Easting 

(cm) 

Bias 

Northing 

(cm) 

Std  Dev 
Easting 
(cm) 

Std  Dev 
Northing 
(cm) 

T-001 

Shotput 

6.3 

9.3 

-5.9 

-8.4 

2.2 

3.9 

T-002 

Small  ISO 

13.0 

10.7 

-9.2 

-5.6 

9.1 

9.1 

T-003 

Small  ISO 

11.3 

13.8 

-10.9 

-9.3 

2.7 

10.2 

T-004 

37  mm 

10.0 

6.8 

-9.6 

-5.2 

2.7 

4.4 

T-005 

75  mm 

9.2 

8.8 

-8.7 

-5.8 

3.0 

6.6 

Average 

10.0 

9.9 

-8.9 

-6.9 

3.9 

6.8 
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Figure  10.  Polarizabilities  extracted  from  27  repeat  measurements  over  the  5  IV S  items. 

6.4  DATAPRODUCTS 

The  MetalMapper  data  collected  as  part  of  this  demonstration  are  available  from  the  ESTCP 
Program  Office.  The  data  were  arranged  into  three  separate  packages  comprising 

(1)  The  field  data  over  the  2,370  targets  identified  at  Pole  Mountain; 

(2)  Measurements  over  the  testpit  (calibration  ball  and  37,  57  and  75  mm  projectiles  as  well 
as  a  stokes  mortar); 

(3)  All  measurements  made  on  the  IYS  strip  (comprising  a  total  of  27  repeats  over  each  of 
the  5  I  VS  items). 

6.4.1  Field  data 

Raw  and  background-corrected  data  in  comma-separated  value  (CSV)  files  are  provided  in 
separate  folders.  The  file  naming  convention  is: 

PM_Sky_NNNNN_Dl  ldddStaticMMMMM_raw_NNNNN.csv  (raw) 

PM_Sky_NNNNN_Dl  ldddStaticMMMMM_bc_NNNNN.csv  (background-corrected) 
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where: 


NNNNN  is  the  anomaly  number 
ddd  is  the  acquisition  day  number 

StaticMMMMM  is  the  MetalMapper  source  .tern  file  name 

Anomaly  numbers  range  from  1  through  2374;  anomaly  numbers  699,  718,  1110  and  1358  were 
not  assigned. 

Background-corrected  files  have  been  normalized  by  the  transmitter  current  (average  of  the  last 
10  values  reported  in  the  raw  CSV  file)  and  have  had  the  background  noise  subtracted  from  the 
data.  During  each  day  of  data  acquisition  a  number  of  measurements  of  the  background  noise 
were  made  at  a  magnetically  quiet  field  site.  One  of  these  measurements  was  used  for  the 
background  correction  of  all  data  from  that  same  day.  The  background  files  are  provided  in  a 
separate  folder. 

Due  to  unresolved  issues  with  our  IMU,  the  heading  obtained  from  the  IMU  was  deemed  to  be 
unreliable.  In  addition,  the  IMU  would  intermittently  cut  out,  resulting  in  no  orientation 
information  for  some  anomalies.  A  handheld  compass  was  used  to  obtain  approximate  heading 
information,  but  compass  readings  were  not  available  for  the  first  two  days  of  data  acquisition. 
The  heading  value  that  appears  in  the  CSV  files  (for  both  raw  and  background-corrected  data) 
represents  the  compass  reading  (corrected  for  local  magnetic  declination)  when  a  compass 
reading  exists.  When  no  compass  reading  is  available,  the  heading  value  is  the  value  obtained 
from  the  IMU  (if  the  IMU  was  functioning).  Information  on  the  availability  of  compass  readings 
and  IMU  status  is  given  in  an  accompanying  file  (PoleMtn_CSV_Log_File.xls)  which  lists,  for 
each  anomaly: 

1 .  Anomaly  number 

2.  Raw  CSV  file  name 

3.  Background-corrected  CSV  file  name 

4.  Background  noise  file  used  for  background  subtraction  in  (3) 

5.  Compass  status:  1  if  a  compass  reading  was  taken;  0  if  a  compass  reading  was  not  taken 

6.  IMU  status:  1  if  the  IMU  provided  orientation  information;  0  if  it  did  not 

6.4.2  Test-pit  measurements 

The  data  format  is  the  same  as  for  the  field-data  files.  A  list  of  the  different  ordnance,  depth  and 
orientation  combinations  collected  are  provided  in  an  accompanying  file 
(PoleMtnCS  V_Log_F  ile_T  estPit.xls) . 

6.4.3  Measurements  on  the  Instrument  Verification  Strip 

The  data  format  is  the  same  as  for  the  field-data  files.  A  list  of  the  different  IVS  items  and  the  27 
repeat  measurements  are  provided  in  an  accompanying  file  (PoleMtn_CSV_Log_File_IVS.xls). 
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6.5  PRODUCTION  TEAM  CLASSIFICATION  METHOD  AND  RESULTS 

The  production  team  independently  fit  polarization  tensor  models  and  developed  a  digging  order 
using  tools  developed  under  ESTCP  MR-201004.  Both  single  and  multi-object  polarization 
tensors  models  were  fit  to  each  anomaly  and  the  fit  results  QC’d  by  the  production  team. 
Ground-truth  information  was  requested  for  32  items  with  three  of  the  ground-truth  anomalies 
corresponding  to  TOI  (Table  7).  Together  with  the  test-pit  items,  the  ground-truth  was  used  to 
determine  dig-list  order  using  the  DigZilla  tool  (Pasion  et  al.,  2012).  The  DigZilla  analysis  used 
all  three  polarizabilities.  An  initial  stop-dig  point  comprising  261  excavations  was  selected  based 
on  visual  review  of  the  data.  All  157  remaining  TOI  were  recovered  with  this  stop-dig  point, 
with  the  last  TOI  recovered  on  the  very  last  dig.  The  production  team  used  the  automated  method 
of  Pasion  et  al.,  (2012)  for  determining  the  stop-dig  point  which  resulted  in  an  additional  74  digs, 
none  of  which  ended  up  being  TOI.  A  further  2  digs  were  added  during  subsequent  QC  and 
neither  of  these  were  TOI  either  (Table  7). 

At  the  final  stop-dig  point  (and  including  groundtruth),  369  anomalies  were  excavated  and  all 
160  TOI  were  identified.  A  total  of  209  of  the  2210  clutter  items  were  excavated:  this  meant  that 
91%  of  the  clutter  could  have  been  left  in  the  ground. 


PoleMtn  Sky  Production  None  MetalMapper  Custom  si  v2  UXO 


No  Difficult  UXOs 


Figure  11.  ROC  curve  for  the  dig-list  submitted  by  the  SKY  production  team. 

Table  7.  Number  of  excavations  and  TOI  recovered  in  each  of  the  4  stages  of  submittals  made  by 
the  SKY  production  team. 


Stage 

Number 

TOI 

TOI 

excavations 

recovered 

remaining 

Training 

32 

3 

157 

Stage  1 

261 

157 

0 

Stage  2 

74 

0 

0 

Stage  3 

2 

0 

0 
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7.0PERFORMANCE  ASSESSMENT 

7.1  PERFORMANCE  CRITERIA 

Table  1  listed  the  performance  criteria  for  the  demonstration.  We  now  list  each  of  the 
performance  objectives  and  determine  if  they  have  been  met. 

7.1.1  Reliability  and  Robustness 
Objective:  General  observations. 

Performance:  Partially  Met. 

With  the  exception  of  the  INS  alignment  errors  during  the  first  part  of  the  project,  the  data 
collection  proceeded  with  few  technical  problems.  Besides  the  INS  alignment,  the  minor  issues 
encountered  were  related  to  a  young  software  platform.  We  expect  issues  such  as  point  ID  not 
recorded  in  the  metadata,  cart  orientation  display  on  map  being  off  by  180  degrees  or  a  lack  of 
units  on  graphs  to  be  worked  out  as  the  software  matures. 

The  hardware  platform  performed  adequately.  Due  to  the  stress  associated  with  the  mounting 
platform,  speed  was  limited  to  2mph  on  rough  terrain,  3mph  on  rough  dirt  roads,  and  5  mph  on 
smooth  dirt  roads.  Going  from  one  point  to  the  next  required  a  minimum  of  three  transmission 
shifts,  with  an  average  of  eight  shifts  per  cued  point.  While  the  existing  platform  can  continue  to 
be  used,  it  does  cause  a  lot  of  extra  wear  and  tear  on  the  vehicle. 

7.1.2  Survey  Rate 
Objective:  200  anomalies  /  day. 

Performance:  Met. 

Table  8  lists  the  number  of  anomalies  that  were  surveyed  each  day.  On  average  205  anomalies 
were  visited  each  survey  day.  The  IMU  related  INS  Alignment  errors  had  a  large  impact  on 
production  rates.  Time  was  lost  both  to  troubleshooting  the  issue  and  in  navigating  to  each 
anomaly  which  was  difficult  without  heading  information.  Neglecting  the  bad  IMU  days,  the 
average  survey  rate  was  32  points  per  hour.  This  rate  does  not  include  GPS  base  stations  setup 
time  or  transportation  time.  The  rate  does  include  down  time  during  the  survey  day  and  time 
spent  on  taking  multiple  shots  of  the  same  point  ID. 

7.1.3  Percentage  of  Assigned  Targets  Completed 

Objective:  100%  as  allowed  by  topography  /  vegetation. 

Performance:  Met,  with  100%  of  targets  surveyed  with  the  MetalMapper. 

Review  of  the  collected  data  reveals  that  100%  of  all  cued-interrogation  anomalies  were 
surveyed.  The  vehicle  platform  was  not  able  to  go  over  obstacles  higher  than  14  inches.  This 
was  not  a  limiting  factor  at  this  site. 
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Table  8  Number  of  Cued-Interrogation  Anomalies  Surveyed  Each  Day 


Day  of 
Year 

Date 

Start 

Cued 

End 

Cued 

Field 

Time 

(hours) 

Number 
of  Point 
IDs 

Number  of 
Recollected 
Points 

Total 

Field 

Shots 

(approx.) 

Point 

IDs/Hour 

Number 
of  Test 
Pit 
Items 

Number 
of  Bad 
headings 
(approx.) 

Other  Notes 

D11201 

7/20/2011 

1840 

1900 

0.3 

2 

3 

5 

6.7 

46 

Test  pit  day. 

D11202 

7/21/2011 

1430 

1600 

1.5 

23 

1 

24 

15.3 

24 

Troubleshoot  INS. 

D11203 

7/22/2011 

812 

1650 

8.6 

211 

7 

218 

24.5 

170 

D11205 

7/24/2011 

825 

1704 

8.5 

186 

9 

195 

21.9 

26 

Start  recording  compass 
headings. 

D11206 

7/25/2011 

812 

1643 

8.5 

201 

5 

206 

23.6 

Unknown  number  of  bad 

IMU. 

D11207 

7/26/2011 

841 

1532 

7 

94 

11 

105 

13.4 

Unknown  (high?)  number  of 
bad  IMU,  bad  positioning 
day,  light  weather. 

D11208 

7/27/2011 

710 

1545 

8.5 

154 

24 

178 

18.1 

Unknown  number  of  bad 

IMU,  repair  fairlead. 

D11209 

7/28/2011 

849 

1633 

7.6 

209 

5 

214 

27.5 

Unknown  (low?)  number  of 
bad  IMU. 

D11211 

7/30/2011 

830 

1420 

6 

162 

3 

165 

27.0 

Troubleshoot  IMU  (DL4  vs 
ProPak),  stable  IMU? 

D11213 

8/1/2011 

740 

1645 

9 

251 

10 

261 

27.9 

Good  IMU. 

D11214 

8/2/2011 

823 

1634 

8.1 

247 

3 

250 

30.5 

Good  IMU. 

D11215 

8/3/2011 

741 

1408 

6.5 

214 

17 

231 

32.9 

Good  IMU,  clutter  field. 

D11216 

8/4/2011 

730 

1630 

9 

316 

10 

326 

35.1 

Good  IMU,  light  weather. 

D11217 

8/5/2011 

800 

944 

1.75 

68 

17 

85 

38.9 

Good  IMU. 
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(a)  EM61  pick  locations  (b)  MetalMapper  center  locations 


(c)  MetalMapper  inversion  locations  (d)  Cumulative  location  errors  (TOI  only) 


Figure  12.  Location  errors  for  (a)  original  EM61  pick  location;  (b)  MetalMapper  center  location 
and  (c)  MetalMapper  inversion  location  with  25  cm  error  circles  shown.  The  cumulative  error  plot 

in  (d)  only  considers  TOI. 

7.1.4  Location  accuracy  of  interpreted  anomalies 

Objective:  MetalMapper  center  location  within  30  cm  of  TOI  at  least  90%  of  the  time. 

Performance:  Effectively  met,  with  the  90th  percentile  occurring  at  31  cm 

Previous  work  with  the  MetalMapper  (Pasion  et  al.,  2012)  has  shown  that  the  ability  to 
accurately  constrain  polarizabilities  degrades  if  an  anomaly  is  not  well  centered  under  the 
MetalMapper  sensor.  Figure  12a  compares  the  EM61  pick  location  to  the  reported  GPS  position 
from  ground-truth.  In  cases  where  more  than  one  item  and  location  was  reported  by  the  dig-team, 
then  the  item  closest  to  the  EM61  pick  location  is  used.  From  inspection  of  this  figure  and  the 
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cumulative  location  error  plot  in  Figure  12d  it  is  evident  that  placing  the  MetalMapper  sensor 
directly  over  the  EM61  pick  location  will  result  in  many  poorly  centered  anomalies. 

At  each  location,  the  moving  arrows  display  was  used  to  center  the  MetalMapper  array.  Figure 
12b  and  d  demonstrate  a  significant  improvement  in  anomaly  centering  relative  to  the  EM61  pick 
locations.  For  TOI  we  did  not  quite  reach  our  stated  performance  objective  of  90%  within  30  cm, 
with  the  90th  percentile  occurring  31  cm.  For  this  analysis  we  retained  all  points  including  those 
with  no  compass  measurement. 

Figure  12c  shows  the  location  error  of  the  polarization  tensor  model  compared  to  the  ground- 
truth  location.  95%  of  all  TOI  were  located  with  an  accuracy  of  better  than  20  cm. 

7.1.5  Maximize  correct  classification  of  munitions. 

Objective:  100%  of  TOI  correctly  identified 
Performance:  Met 

The  scoring  analysis  conducted  by  IDA  indicated  that  all  160  TOI  were  found  before  the 
designated  stop-digging  point. 

7.1.6  Maximize  correct  classification  of  non-munitions. 

Objective:  Greater  than  75%  reduction  in  non-TOI  while  retaining  all  TOI 
Performance:  Met 

At  the  selected  operating  point,  91%  of  the  non-TOI  were  left  in  the  ground. 

7.1.7  Specification  of  no-dig  threshold. 

Objective:  All  TOI  recovered  with  75%  of  non-TOI  left  in  the  ground 
Performance:  Met 

At  the  selected  operating  point,  all  TOI  were  recovered  with  91%  of  non-TOI  left  in  the  ground. 

7.1.8  Minimize  number  of  anomalies  that  cannot  be  analyzed. 

Objective:  Reliable  target  parameters  can  be  estimated  for  >  90%  of  anomalies  on  each  sensor’s 
detection  list. 

Performance:  Met 

In  the  final  interpretation,  an  acceptable  model  was  fit  to  every  single  anomaly,  so  that  there 
were  zero  “can’t  analyze”  anomalies.  To  achieve  this  metric  required  rigorous  daily  QC  and  the 
recollection  of  125  anomalies:  or  5.2%  of  the  total  number  of  anomalies.  Without  this  investment 
in  QC  and  recollects,  we  would  still  have  met  the  “can’t  analyze”  metric  of  <  10%. 
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8.0COST  ASSESSMENT 

8.1  COST  REPORTING 

Cost  categories  for  this  demonstration  are  mobilization,  field  survey,  data  analysis, 
demobilization,  and  reporting.  These  costs  were  tracked  throughout  the  demonstration  and  are 
presented  in  Table  9  (fully  burdened  costs  for  the  demonstration  and  the  pre-mobilization  tests 
conducted  in  Denver). 


Table  9  Fully  Burdened  Costs 


Categories 

Pre-mob 

testing 

Mobilization 

Data 

collection 

Data 

processing 

Reporting 

Total 

(excluding 

testing) 

Total 

Labor 

$  52,275 

$  31,621 

$  53,205 

$  9,208 

$  16,156 

$162,465 

Equipment 

$  8,478 

$  7,869 

$  34,534 

$  - 

$  - 

$  50,882 

Travel 

$  4,162 

$  1,148 

$  5,872 

$  - 

$  342 

$  11,524 

Matls  and  Supplies 

$  91 

$  550 

$ 

738 

$ 

$ 

$  1,379 

Std  ODCs 

$  2,798 

$  2,245 

$  452 

$  929 

$  9,825 

Total 

$  67,805 

$  97,750 

$  9,659 

$  17,428 

$236,075 

8.2  COST  ANALYSIS 

During  the  Pole  Mountain  demonstration  MetalMapper  data  were  collected  at  2,370  anomaly 
locations.  Based  on  the  cost  information  presented  in  Table  9  the  per  anomaly  costs  (excluding 
mobilization  and  reporting)  were 

•  $41.24  for  data  collection;  and 

•  $4.08  for  data  processing. 

The  total  cost  per  anomaly  (for  data  collection  and  processing)  was  $45.52. 

Using  an  often  quoted  rule  of  thumb  that  each  excavation  costs  $100,  then  without  deploying  the 
MetalMapper  system  the  excavation  costs  would  have  been  approximately  $237,000.  With  the 
MetalMapper  only  369  anomalies  needed  to  be  excavated  at  a  cost  of  $36,900.  When  combined 
with  the  MetalMapper  mobilization,  data  collection,  processing  and  reporting  costs  of  $168,270 
the  cost  of  clearance  using  the  MetalMapper  was  $205,170:  a  saving  of  $31,830  (or  13%). 
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